{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "## Neural networks\n\nNow that we know what neurons are, we are ready for the final step: the neural network!. A neural network is literally made out of a network of neurons that are connected together. \n\nSo far, we have just looked at single neurons, that only have a single output.\nWhat if we want multiple outputs?\n\n\n### Multiple output models\n\nWhat if we wanted to distinguish between apples, bananas, *and* grapes? We could use *vectors* of `0` or `1` values to symbolize each output.\n\n<img src=\"data/fruit-salad.png\" alt=\"Drawing\" style=\"width: 300px;\"/>\n\nThe idea of using vectors is that different directions in the space of outputs encode information about different types of inputs."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we extend our previous model to give multiple outputs by repeating it with different weights. For the first element of the array we'd use:\n\n$$\\sigma(x;w^{(1)},b^{(1)}) := \\frac{1}{1 + \\exp(-w^{(1)} \\cdot x + b^{(1)})};$$\n\nthen for the second we'd use\n\n$$\\sigma(x;w^{(2)},b^{(2)}) := \\frac{1}{1 + \\exp(-w^{(2)} \\cdot x + b^{(2)})};$$\n\nand if you wanted $n$ outputs, you'd have for each one\n\n$$\\sigma(x;w^{(i)},b^{(i)}) := \\frac{1}{1 + \\exp(-w^{(i)} \\cdot x + b^{(i)})}.$$"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Notice that these equations are all the same, except for the parameters, so we can write this model more succinctly, as follows. Let's write $b$ in an array:\n\n$$b=\\left[\\begin{array}{c}\nb_{1}\\\\\nb_{2}\\\\\n\\vdots\\\\\nb_{n}\n\\end{array}\\right]$$\n\nand put our array of weights as a matrix:\n\n$$ \\mathsf{W}=\\left[\\begin{array}{c}\n\\\\\n\\\\\n\\\\\n\\\\\n\\end{array}\\begin{array}{cccc}\nw_{1}^{(1)} & w_{2}^{(1)} & \\ldots & w_{n}^{(1)}\\\\\nw_{1}^{(2)} & w_{2}^{(2)} & \\ldots & w_{n}^{(2)}\\\\\n\\vdots & \\vdots &  & \\vdots\\\\\nw_{1}^{(n)} & w_{2}^{(n)} & \\ldots & w_{n}^{(n)}\n\\end{array}\\right]\n$$\n\nWe can write this all in one line as:\n\n$$\\sigma(x;w,b)= \\left[\\begin{array}{c}\n\\sigma^{(1)}\\\\\n\\sigma^{(2)}\\\\\n\\vdots\\\\\n\\sigma^{(n)}\n\\end{array}\\right] = \\frac{1}{1 + \\exp(-\\mathsf{W} x + b)}$$\n\n$\\mathsf{W} x$ is the operation called \"matrix multiplication\"\n\n[Show small matrix multiplication]\n\nIt takes each column of weights and does the dot product against $x$ (remember, that's how $\\sigma^{(i)}$ was defined) and spits out a vector from doing that with each column. The result is a vector, which makes this version of the function give a vector of outputs which we can use to encode larger set of choices. \n\nMatrix multiplication is also interesting since **GPUs (Graphics Processing Units, i.e. graphics cards) are basically just matrix multiplication machines**, which means that by writing the equation this way, the result can be calculated really fast."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "This \"multiple input and multiple output\" version of the sigmoid function is known as a *layer of neurons*.\n\nPreviously we worked with a single neuron, which we visualized as\n\n<img src=\"data/single-neuron.png\" alt=\"Drawing\" style=\"width: 300px;\"/>\n\nwhere we have two pieces of data (green) coming into a single neuron (pink) that returned a single output. We could use this single output to do binary classification - to identify an image of a fruit as `1`, meaning banana or as `0`, meaning not a banana (or an apple).\n\nTo do non-binary classification, we can use a layer of neurons, which we can visualize as\n\n<img src=\"data/single-layer.png\" alt=\"Drawing\" style=\"width: 300px;\"/>\n\nWe now have stacked a bunch of neurons on top of each other to hopefully work together and train to output results of more complicated features. \n\nWe still have two input pieces of data, but now have several neurons, each of which produces an output for a given binary classification: \n* neuron 1: \"is it an apple?\"\n* neuron 2: \"is it a banana?\"\n* neuron 3: \"is it a grape?\""
      ],
      "metadata": {}
    }
  ],
  "nbformat_minor": 2,
  "metadata": {
    "language_info": {
      "file_extension": ".jl",
      "mimetype": "application/julia",
      "name": "julia",
      "version": "0.6.2"
    },
    "kernelspec": {
      "name": "julia-0.6",
      "display_name": "Julia 0.6.2",
      "language": "julia"
    }
  },
  "nbformat": 4
}
